A design of on-line analytical processing (OLAP) aims to satisfy specific query and reporting requirements in a decision support or multi-dimensional environment. A data warehouse and the OLAP are important supporting technologies of enterprise-level decision supporting.
However, the performance of the OLAP over massive data is hard to satisfy the gradually increasing requirements on real-time high-performance analytical processing. The bottleneck of the performance mainly lies in the following two aspects: one is low hard disk I/O performance, and the other is insufficiently optimized query processing technology. A mainstream technology of improving the hard disk I/O performance is column store. The column store can realize storage and access of data according to columns. Recently, the query processing technology adopted by a column store database is mainly classified into two types: one is the column data query processing technology, and the other is that column store data is dynamically converted into row store data through a cache of data in a memory, and a conventional row store query processing technology is adopted.
The column data query processing technology refines relational algebra into column algebra, and divides the relational operation into several ordered column operations. The column data query processing technology requires a large number of materialized join indexes and additional cost for join operations between columns. In the OLAP query, a foreign key column in the fact table needs to be scanned twice to complete filtering and group-by operations, which increases additional I/O access overhead.
In the dynamic column/row conversion query processing technology widely adopted currently, accessing data according to columns improves the I/O efficiency of the projection operation; however, a pipeline query processing technology of row store is still adopted in the level of query processing. The pipeline query processing technology needs to first materialize join data required by the query, and when the selectivity is low, a large amount of materialized data is finally abandoned, which consumes additional memory bandwidth resources and degrades the processing performance of the memory.
The parallel OLAP is mainly applied in a main memory database. In a row store database, a representative technology is materializing multiple tables into one denormalized join table by using a materialized join method, performing parallel processing on distributed aggregate functions on the basis of a horizontal fragmentation mode, and merging aggregate result sets. This technology reduces the complexity of the query processing and increases the parallel speedup ratio, but requires larger space. On the column store model, a representative technology is performing logical horizontal fragmentation on the join operation between columns having large execution cost, performing parallel join, and merging parallel join results for the subsequent column operation. This technology has simple implementation procedure, but has higher workload proportion of serial processing, and overall parallel processing resources are not fully utilized.
In the application scenario of a disk resident database, the processing performance of the parallel OLAP is limited by the I/O performance of parallel disk access, and the parallel processing technology based on the horizontal fragmentation mode is rarely adopted, instead, a concurrent query processing technology of a shared disk scan mode is adopted. The core of the concurrent query processing technology is minimizing the memory processing time of the OLAP, so that as many concurrent query tasks as possible can be processed in one I/O latency. The conventional hash join generates a large number of hash tables in the concurrent query processing load, and the hash tables will exhaust the memory resources of the system, thereby causing poor performance of the hash join. The selectivity of the OLAP query is relatively high, and therefore, the concurrent query processing faces a larger performance bottleneck, and it is difficult to obtain desired performance in practice.